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ABSTRACT 

The  very  large  varieties  of  the  multi-sensor  scene  signatures  that  needs  to  be  considered  for  building  a  robust 
machine-based  scene  analysis  system  requires  that  one  explores  the  possibility  of  deriving  multi-sensor 
representations  that  are  at  the  least  invariant  to  scale,  translation,  and  rotation  with  respect  to  the  observer. 

In  this  paper  we  develop  methodologies  for  multi-sensor  scene  analysis  based  on  the  theory  of  invariant  algebra. 
Two  cases  of  similar  and  dissimilar  sensor  types  are  considered.  The  fused  invariant  for  the  case  of  similar  sensor 
types  are  unchanged  under  material  as  well  as  under  affine  transformations.  For  the  case  of  dissimilar  sensors  the 
approach  leads  to  the  derivation  of  the  multi-sensor  invariant  expressions  that  remain  unchanged  under  scale, 
translation  and  rotation  with  respect  to  the  observer.  Examples  of  the  application  of  the  approach  on  real  multi¬ 
sensor  data  are  presented. 

1.  INTRODUCTION 

The  large  varieties  of  the  multi-sensor  scene  signatures  that  needs  to  be  considered  for  building  a  robust  machine- 
based  scene  analysis  system  requires  that  one  explores  the  possibility  of  deriving  multi-sensor  representations  that 
are  at  the  least  invariant  to  scale,  translation,  and  rotation  with  respect  to  the  observer. 

In  this  paper  we  develop  methodologies  for  multi-sensor  scene  analysis  based  on  the  theory  of  invariant  algebra. 
Two  cases  of  similar  and  dissimilar  sensor  types  are  considered.  The  fused  invariant  for  the  case  of  similar  sensor 
types  are  unchanged  under  material  as  well  as  under  affine  transformations.  For  the  case  of  dissimilar  sensors  the 
approach  leads  to  the  derivation  of  the  multi-sensor  invariant  expressions  that  remain  unchanged  under  scale, 
translation  and  rotation  with  respect  to  the  observer. 

In  the  Case  I  the  sensor  types  are  assumed  to  be  similar,  correlated,  co-located  and  registered.  Hyperspectral  images 
are  good  example  of  this  Case.  Case  II  deals  with  dissimilar  sensor  types  where  the  imagery  can  be  assumed  to  be 
uncorrelated.  Laser  range,  passive  IR  ,  and  Doppler  imaging  sensors  are  examples  of  this  second  case. 

2.  CASE  I :  SIMILAR  SENSORS 

From  the  Planck  Law  one  has  the  following  relationship  between  the  emissivity,  temperature,  wavelength, 
and  the  spectral  radiant  emittance  W^. 

w  _  2;dic^  (1) 

X  iS  ch 
^  gAKT 


Where  T  is  temperature  in  degree  Kelvin,  is  spectral  emissivity,  h  is  the  Planck’s  constant,  and  X  is  the 
wavelength-  In  (1)  only  emissivity  is  material  dependent. 
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The  scene  radiation,  Rn(j,k)  where  n  is  the  index  indicating  a  particular  spectral  band  is  obtained  by  integrating  (1) 
over  different  bands  of  frequencies.  From  a  wide  range  of  frequency  bands  (different  n  values)  different  spectral 
images  corresponding  to  the  same  scene  are  obtained. 

When  the  effects  of  the  radiation  reflectance  are  negligible  or  ignored  and  we  assume  that  the  scene  is  in  thermal 
equilibrium  the  radiation  varies  only  with  (and  frequency)  in  a  particular  scene.  The  thermal  equilibrium 
assumption  is  useful  because  target  detection  and  classification,  under  this  assumption  cannot  be  performed  only  by 
exploiting  the  temperature  differences  between  targets  and  background.  The  variations  of  with  frequency  for 
different  materials  and  paints  are  well  documented  [1].  Table  1  shows  typical  emissivity  values  of  several  material 
averaged  over  all  wavelengths  at  20°  C. 


TABLE  I.  Typical  Emissivity  of  Various  Material 


Material 

Emissivity 

Water 

0.96 

Wet  Soil 

0.95 

Concrete 

0.92 

Steel 

016 

Wood 

0.90 

Glass 

0.94 

When  the  effects  of  radiation  reflectance  can  not  be  ignored  the  thermal  equilibrium  assumption  is  no  longer  useful 
for  scene  analysis. 

The  out  p  ut  of  a  focal  plane  array  (FPA),  in  general,  is  a  linear  function  of  the  incidence  photons  [6]: 

N„  =K„^^:,(A){e,(l,j)WyT,)  +  [l-^,(l,J)]W,(T,)}dA  +  NJ  (2) 

Where  Ny  is  the  total  number  of  accumulated  electrons  at  pixel  ij,  Ky  is  a  coefficient  that  is  dependent  on  the  active 
pixel  area,  optical  transmission,,  frame  time,  and  pixel  angular  displacement  from  the  optical  axis,  and  the  f-number  of 
the  optics.  The  quantum  efficiency  of  the  ij  pixel  is  denoted  by  ^ij(A) ,  and  background  radiant  reflectance  is  shown 

by  (Tj, )  .  Tj,  is  the  background  temperature  in  degrees  Kelvin,  and  Xi  and  X2  deflne  the  spectral  band  of  the 
sensor.  Finally,  the  dark  charge  for  the  pixel  ij  is  denoted  by  Ny .  The  eJ^^(i,j)  indicates  the  spectral  emissivity  at  the 
pixel  location  ij. 

The  effect  of  the  spectral  emissivity-  and  its  variations  with  frequency  for  different  materials-can  be  seen  in  the 
equation  (2). 

For  each  pixel  e(i,j),  an  n-dimensional  vector  (n  being  the  number  of  frequencies  used),  consider  the  probability  of  it 
being  from  a  material  71^ ,  k  being  the  number  of  different  materials  in  the  scene,  be  denoted  as  p(e|7r  )  and  the 
probability  of  material  occurrence  71^  as  p(7i^).  Then  according  to  the  Bayes  Decision  rule  one  has  to  select  the 
following: 

i,  P(e|7rk)p(7fk)  ^ , 

max{p(;rke)  = - - ;Vk}  (3) 

k  p(e) 

p  (  e  I TT  Ij  )  is  assumed  to  be  known  for  each  frequency  k  at  a  range  of  temperatures  of  interest.  This  assumption  is 
not  restrictive  since  for  different  material  (and  paints)  the  emissivity  as  function  of  frequency  and  temperature  has 
been  documented  [1].  The  p(e)  is  obtained  from  the  following: 


(4) 


p(e)=  Sp(e|7rk)p(7rJ 

Once  for  each  pixel  a  material  label  has  been  chosen  a  new  image  is  formed.  This  image  is  formed  by  replacing  the 
value  of  each  pixel  wit  its  most  likely  emissivity  label  (iron=l,  water=2,  etc.).  Denoting  each  pixel  as  %(i,j)  the 
information  content  of  the  image  varies  by  the  frequency  of  occurrence  of  the  emissivity  label  in  the  image. 


3.  MOMENT  GENERATING  EUNCTION  AND  INVARIANT  ALGEBRA 

If  the  joint  characteristic  function  of  random  variables  x  and  y  has  a  Taylor-series  expansion  valid  in  some  region 
abo  ut  the  origin,  it  is  uniquely  determined  in  this  region  by  the  moments  of  the  random  variables  [2]  : 


M,,(U,V)=  E  E  E(x"y‘“) 


n„m,(u)"(v)” 


n=0m=0 


n!m! 


(5) 


Hence  knowing  M,,y(u,v)  would  imply  knowing  7C]f(x,y),  the  joint  probability  density  function  of  the  x  and  y  random 

variables.  This  implies  that  one  can  represent  the  probability  density  function  denoting  an  image  in  terms  of  a  bi¬ 
variate  homogenous  polynomial  referred  to  as  a  binary  form  in  terms  of  the  image  statistical  moments.  The  changes 
in  the  orientation,  scale  or  position  in  the  field  of  view  of  the  objects  in  this  new  image,  or  in  other  words  the  change 
of  the  coordinate  system  will  transform  the  probability  density  function  %(x,y)  accordingly. 

What  remains  unchanged  under  these  transformations  fall  under  the  domain  of  theory  of  invariant  algebra.  The 
objectives  pursued  under  this  branch  of  algebra,  developed  in  the  19th  Century  by  Cayley  and  Silvester,  is  the  study 
of  the  algebraic  expressions  that  remain  unchanged  under  linear  transformation  of  the  coordinate  systems  [3,4]. 

If  a  binary  p-ic  (polynomial  of  two  variables  and  order  p)  has  an  invariant 


then  the  moments  of  order  p  has  an  algebraic  invariant 


f(M'p0’---’M'0p)^l‘^l'^*^(Pp0v-5P0p) 

where  J  is  the  Jacobian  of  the  transformation  A  and  O)  are  constants. 

Using  the  tools  of  invariant  algebra  the  following  seven  expressions  that  are  invariant  under  size,  rotation  and 
translation  can  be  obtained  [5]. 

3.1.  Second  and  Third  Order  Invariants 

Defining  the  central  moment  of  order  p-l-q  as: 


Jt pq  =  X  X  (x -x)P  (y  -  y)'^  ;'r  k  (x,  y) 

X  y 

The  normalized  central  moments  are: 

Z7pq(^k)= 

/too  +1 

The  first  and  the  second  Order  Invariants  are  then 


(8) 


(9) 


<l>l(^k)  =  V20  +V02 


(10) 


02(^r)  =  (n2O  -n02)^  +4n?i)  (11) 

03  =  (V30-^Vnf  +OV2i-no3f 

04  =(^30-^12)^+(^21+^?03)^ 

05  =  (^30  -3t7i2)(^30  +^?12)[(^?30  +^?12)^  -3(r?21  +^03^  1  +  0^21  -?7o3)(?721  +??03)) 

[3(^730  +^12)^  +(^721-^?03)^] 

06  =  (V20-nQ2)i(n30+nn  f  -(^?21  +  n03)^]  +  4rji,(rj30  +  rj,2)(r]2i  +^103^ 


07(^k)  =  (3??2i  -T?3o)(^3()+f?i2)[(^3o+r7,2)^  -3(rj2i  +rjo3)^] 

+(3^?12  -^73o)(^21  +^03)[3(^?30  +^12)^  “(^21  +^03)^] 


4.  INVARIANCY  TO  MATERIAL  CHANGES 


(12) 


For  the  Case  I,  the  above  functions  are  l)themselves  functions  of  the  material  that  the  targets  and  scene  are  made  of 
and  2)  are  k-ary  forms.  Consequently  for  any  linear  transformation  in  JC]^  (changes  in  material)  there  exist  a  set  of 
invariant  expressions  that  will  remain  unchanged.  These  Second  Order  Invariants  will  be  Invariants  under  scene 
rotation,  scale,  translation  and  material  transformations.  The  expressions  (j)j(%)  to  (j)7(%)  are  polynomials  of  order 

1  to  4  in  terms  of  for  various  k.  Each  of  different  k  values  indicate  a  different  material.  Any  linear  transformation 
of  Jt]j  indicates  a  change  of  material  in  the  scene  such  as  changing  the  paint  on  a  target,  or  having  the  objects  on  a  dry 
land  verses  wet  land,  or  for  a  target  being  on  a  grass  verses  being  on  a  concrete  background. 

5.  CASE  11:  DISSIMILAR  SENSOR 

For  this  case  the  output  of  the  sensors  can  be  assumed  to  be  non-correlated.  This  is  valid  for  the  case  of  colocated, 
registered  passive  IR ,  Laser  range,  and  Doppler  imaging  sensors. 

One  can  assume  that  in  this  case  each  image  represents  a  conditional  density  function  p(f(x,  y)|  A) .  Where 

X  represents  a  particular  imaging  sensor  output.  Then  the  effect  of  using  all  of  the  sensory  outputs  is  equivalent  to  the 
use  of  the  total  probability  function.  The  total  probability  density  function  can  be  obtained  by: 

P(f(x,y))  =  Ip(f(x,y)|A)p(A)  (13) 

A 


The  fused  invariant  expressions  then  can  be  extracted  from  this  total  probability  function. 

Figures  1-3  show  the  co-located,  and  registered  outputs  of  a  particular  laser,  Doppler  and  passive  IR  imaging 
sensors  obtained  from  the  US  Army’s  Night  Vision  and  Electro-Optical  Center.  The  Table  II  shows  their 
corresponding  seven  invariant  expressions.  These  invariants  can  be  seen  to  be  different  for  different  sensory  outputs. 


Figure  1 .  Laser  Range  Image  Figure  2.  Doppler  Image 


Figure  3.  Passive  IR  Image 


Figures  4-7  show  the  total  probability  image  at  four  different  orientations.  Their  corresponding  invariant  expressions 
are  summarized  in  the  Table  III.  As  can  be  seen  the  fused  invariants  remain  as  expected  unchanged  under  the 
orientation  changes. 


TABLE  11.  Invariants  of  the  Conditional  Probabilities 


INVARIANTS 

RANGE 

DOPPLER 

IR 

12381.447771 

13568.759615 

13354.107680 

89364967.289028 

65854066.127031 

62477079.926206 

03 

1214.153333 

0.287704 

99.834628 

04 

1679.326671 

0.348603 

50.435774 

05 

-2175770.238461 

0.085832 

3499.153708 

06 

863328.796422 

2824.946900 

348173.107680 

07 

554695.407939 

-0.095781 

-812.440325 

Figure  4.  Total  Probability  Image 


Figure  5.  Total  Probability  Image 
1 80  Degrees  Rotated 


Figure  6.  Total  Probability  Image 
Mirrored  Around  Vertical  Axes 


TABLE  III.  Invariants  of  the  Total  Probability  at  Four  Different  Orientations 


INVARIANTS 

Original  Position 

90  Degrees 
Rotation 

180  Degrees 
Rotation 

Mirror  around 
Vertical  Axes 

12736.175825 

12736.175825 

12736.175825 

12736.175825 

81187446.960661 

81187446.960661 

81187446.960661 

81187446.960661 

m 

1070.623457 

1070.623457 

1070.623457 

1070.623457 

04 

1162.027092 

1162.027092 

1162.027092 

1162.027092 

05 

-1204674.097088 

-1204674.097088 

-1204674.097088 

-1204674.097088 

06 

-2936904.647600 

-2936904.647600 

-2936904.647600 

-2936904.647600 

07 

531810.313835 

531810.313835 

531810.313835 

531810.313835 

The  testing  of  the  presented  procedure  to  answer  the  issues  of  computational  complexities  verses  storage 
requirements  needed  in  general  model/template  techniques  is  currently  underway.  It  is  however  an  accepted  fact  that 
one  can  not  store  signatures  at  all  conceivable  variations  in  scene  and  target  geometry  and  for  every  variations  in 
their  material  contents.  The  procedures  presented  here  are  attempts  to  address  the  combinatorial  explosive  problem 
of  signature  storage  without  sacrificing  the  target  classification  performance  requirements  The  number  of  Invariants 
needed  for  scene/target  representation  is  a  function  of  the  target  types  and  scene  complexity.  For  example  for 
separating  targets  such  as  tank  versus  truck  the  second  and  third  order  Invariants  may  be  sufficient.  However 
discriminating  among  similar  type  targets  such  as  tank  versus  tank,  requires  higher  order  Invariants. 

6.  SUMMARY 


In  this  paper  a  methodology  for  multi-sensor  scene  analysis,  based  on  the  theory  of  invariant  algebra  is  presented. 


Two  different  cases  are  considered.  For  the  case  of  similar  sensors  such  as  hyperspectral  imaging  sensors  the 
methodology  leads  to  the  representations  that  are  Invariants  to  scene  material  s  as  well  as  scale,  rotation,  and 
translation  with  respect  to  the  observer. 


For  the  case  of  dissimilar  imaging  sensors,  such  as  laser  range,  Doppler  and  passive  IR  the  presented 
methodology  leads  to  the  derivation  of  sensor  fused  Invariants. 

The  use  of  these  approaches  can  substantially  reduce  the  number  of  frames/templates  needed  for  multi-sensor/multi- 
spectral  scene  analysis,  and  target  classification  by  their  exploitation  of  the  scene/  target  invariant  properties. 
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